[NVBug: 6563509] Drop Phi-3-vision / Phi-4-multimodal PTQ support - #2115

how about we make this a util function?

Done in 8444c11 — extracted as _build_meta_skeleton(), placed next to the other get_model helpers (_resolve_init_config / _get_config_dtype). get_model is now a single call, and the fallback is unit-testable on its own: verified all three paths (tier-1 succeeds / tier-1 fails on meta and tier-2 succeeds / both fail returning None with one warning).

Follow-up: _build_meta_skeleton() simplified further in 1b062da. The two-tier retry is gone — it is now a single include_buffers=False build, because that is what Transformers' own from_pretrained uses and the old include_buffers=True probe was stricter than the loader it was predicting. Details in the design thread.

Final state on this one: the helper you asked me to extract has now been deleted outright in 4e0157d. Sizing the device map from the checkpoint index removes the need to construct a model at all, so _build_meta_skeleton and everything that fed it (_resolve_init_config, the from_config resolution, the accelerate imports) are gone. Net 74 lines lighter than before the extraction.

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

no need to explain here.

Done in 8444c11 — note removed. The CHANGELOG entry under 0.46 Backward Breaking Changes carries the explanation.

Bot comment.

The supplied evidence does not establish that Phi-3-vision is unloadable across the supported range. The dependency floor and CI matrix explicitly include Transformers 4.57, while the listed Phi-3 blocker (_tied_weights_keys as a list) is described as a Transformers 5.x failure; the version matrix in the PR body only tests Phi-4-MM. Since this is a breaking support removal, please provide/test the Phi-3-vision failure on 4.57 (and identify its blocker), or retain Phi-3-vision support and scope the deletion/documentation to Phi-4-MM.

Fair reading of the evidence, but we are keeping Phi-3-vision in the removal.

You are right about what was and was not proven. Phi-3-vision has no get_peft_model call, so it does not have the transformers-4.52 blocker — that one is Phi-4-MM only. The defect I verified for it is _tied_weights_keys declared as a list (modeling_phi3_v.py:1214), which is a Transformers 5.x failure, and I did not run it end-to-end on 4.57.

We are dropping it anyway, for product reasons rather than a 4.57 repro: Phi-3-vision is the older, superseded model, and Phi-4-multimodal is its replacement. If we cannot support the successor there is no reason to carry the predecessor — it is strictly the weaker model and nobody starting today would pick it over Phi-4-MM.

The technical direction agrees. Phi-3-vision is confirmed broken on Transformers 5.x, and per the 0.46 changelog we already bumped the floor to 4.57 with "Transformers 4.x support will be dropped in a future release". So the best case for keeping it is a support window that closes on its own within a release or two, on a model whose remote code Microsoft is not updating. The same reasoning already applied to VILA / NVILA in this release.

Happy to reopen if someone has an active Phi-3-vision workflow on 4.57.

Bot comment.

This addresses the implementation shape, but the prior critical test request is still unresolved: none of the changed files adds tests for this helper. Please commit mocked coverage for (1) include_buffers=True succeeding, (2) the first attempt failing and include_buffers=False succeeding, and (3) both attempts failing with one warning. The last case should also exercise get_model to pin that infer_auto_device_map is skipped while from_pretrained is still called. Existing test_get_model_* tests only exercise a successful skeleton.

Both points addressed in 1b062da — and you were right that the retry needed justifying. It did not survive the justification, so it is gone.

Design. The probe answers exactly one boolean: will the model spill to CPU, so should max_memory get the gpu_max_mem_percentage haircut. (inferred_device_map has two references in the file — it is built, tested with "cpu" in .values(), and discarded.) infer_auto_device_map reaches that via compute_module_sizes, which is tensor.numel() * dtype_byte_size(tensor.dtype) — shapes and dtypes, never storage.

So the probe only has to reproduce the module tree, and the binding constraint is that it must be no stricter than the loader it predicts:

include_buffers=True was stricter than both. accelerate implements it as a bare global device context, so it captures scratch arithmetic in __init__ that has nothing to do with weights. Transformers added meta_device_safe_creation_ops precisely because pre-v5 remote code derives scalars that way. A model doing int(torch.tensor(...)) in __init__ loads fine via from_pretrained on 4.x and 5.x, but died in our probe — an optimization killing runs it exists only to speed up.

So the answer to "why not just use the existing include_buffers=False pattern" is: we should, and now do. Single build, no retry loop, no noqa: PERF203, no error accumulation. Measured cost on real checkpoints is nil — Qwen3-8B and DeepSeek-R1-Distill-Llama-70B both retain 0.0 MiB and produce identical sized totals (15.256 / 131.417 GiB) either way, for +0.09s and +0.19s.

One note on build_meta_causal_lm: it uses include_buffers=False for a different reason, not as a general preference. Its skeleton is kept and has real weights loaded into it under FSDP2, and inv_freq / original_inv_freq are persistent=False (verified absent from state_dict()), so they are computed at init and never loaded — on meta they would stay meta. It needs real buffers; this probe needs none. Same call, opposite motivation.

Tests. Added in tests/examples/hf_ptq/test_example_utils.py: the probe uses permissive patching, it survives a meta-hostile __init__ (int(torch.tensor(...)), the Phi-4-MM shape), it returns None with a warning on failure, and get_model skips infer_auto_device_map while still calling from_pretrained — asserting no max_memory cap is invented, per the CodeRabbit thread. 32 passed in the file.

Follow-up: the helper this thread was about no longer exists as of 4e0157d. _build_meta_skeleton and its four tests are gone — the device map is now sized from the checkpoint on disk, so no model is constructed and there is no fallback path to cover. The replacement _checkpoint_size_bytes has its own tests (index metadata, shard-sum fallback, malformed index) plus get_model tests for capping only when the weights exceed the GPU budget, no cap when the size is unknown, and an assertion that nothing is constructed during sizing.

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

Bot comment.

This still says Phi-3-vision “no longer loads on any version ModelOpt supports,” but the author’s reply confirms it was not tested on supported Transformers 4.57 and that its identified _tied_weights_keys blocker is a 5.x failure. If the removal is a product decision because the model is superseded, document that rationale instead of making an unverified compatibility claim (or provide the 4.57 repro/blocker).

Addressed in 17dfd3b — you were right that the entry was making an unverified claim.

The changelog now splits the two rationales. Phi-4-multimodal is described as unloadable on any supported version, which the matrix does support. Phi-3-vision is described as a supersession removal: "dropped alongside it: it is the older, superseded model in the same family, so with its successor unsupportable there is no reason to keep carrying the predecessor." A parenthetical records what was actually verified — it shares the list-valued _tied_weights_keys defect so is broken on Transformers 5.x, but does not hit the peft blocker. No 4.57 claim is made for it. The PR description carries the same split.

Bot comment.

The fallback warning was corrected, but this changelog entry still uses the nonexistent --gpu_max_memory_percentage spelling and implies that flag independently covers the lost estimate. The actual option is --gpu_max_mem_percentage, and in this failure path it only takes effect with --use_seq_device_map; please mirror the accurate warning text here.

Addressed — first in 1b062da, and the entry has since been rewritten again in 4e0157d.

You were right on both parts: the changelog still carried the nonexistent --gpu_max_memory_percentage spelling, and it implied that flag alone covered the lost estimate when in this path it only takes effect together with --use_seq_device_map.

That text is now gone entirely rather than corrected, because the fallback it described no longer exists: the device map is sized from the checkpoint on disk instead of from a constructed model, so there is no skeleton to fail and no estimate to lose. The entry now describes that mechanism, and the only remaining degradation is "size unreadable → no cap applied", the same as before.

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

Bot comment.

This prior compatibility comment is still unresolved. The author confirmed Phi-3-vision was not tested on supported Transformers 4.57 and that its identified _tied_weights_keys failure applies to 5.x, so “no longer loads on any version ModelOpt supports” remains unsupported for that model. Please describe Phi-3-vision as a product/supersession removal (using the rationale from the author's reply), or provide the missing 4.57 repro/blocker. Keep the technical incompatibility claim scoped to Phi-4-MM where the version matrix supports it.

Addressed in 17dfd3b — you were right that the entry was making an unverified claim.

The changelog now splits the two rationales. Phi-4-multimodal is described as unloadable on any supported version, which the matrix does support. Phi-3-vision is described as a supersession removal: "dropped alongside it: it is the older, superseded model in the same family, so with its successor unsupportable there is no reason to keep carrying the predecessor." A parenthetical records what was actually verified — it shares the list-valued _tied_weights_keys defect so is broken on Transformers 5.x, but does not hit the peft blocker. No 4.57 claim is made for it. The PR description carries the same split.

Bot comment.

[P1] Please release this parameter-only skeleton before loading the checkpoint. With include_buffers=False, parameters are meta tensors, but registered buffers and other tensors retained by the constructor are real CPU allocations. This local remains referenced while the right-hand side of the later model = auto_model_module.from_pretrained(...) materializes the actual checkpoint, so the two allocations overlap; disk-offload also builds the skeleton even though that branch never calls infer_auto_device_map. For models with large constructor buffers or caches, this can bypass max_cpu_memory_gb and OOM the memory-constrained path. Please skip skeleton construction in disk-offload mode and release model after device-map inference, before from_pretrained().

Both points were correct, and they pushed the design somewhere better — addressed in 4e0157d, though not the way you suggested.

You were right on both counts against the code as it stood: Python evaluates the right-hand side before rebinding, so model = auto_model_module.from_pretrained(...) kept the probe alive for the whole checkpoint load, and with include_buffers=False its buffers are real CPU allocations. And the _disk_offload branch never touched the probe, so it was paying for one unused in exactly the memory-constrained path.

Rather than release the probe and skip it in disk-offload, the probe is gone. It only ever produced one boolean — will the weights fit on the GPUs, so should max_memory get the haircut — and that number is already on disk. metadata.total_size in the safetensors index equals what compute_module_sizes computed on the constructed model, exactly: Qwen3-8B 16381470720 B = 15.256 GiB, DeepSeek-R1-Distill-Llama-70B 141107412992 B = 131.417 GiB, both matching the meta model.

So _checkpoint_size_bytes() reads the index (falling back to summing shards, None if unreadable), and no model is constructed during device-map selection. That removes the overlap you flagged by construction, removes the disk-offload waste, and removes the original NVBug crash site entirely rather than tolerating it — Phi-4-multimodal now sizes at 10.383 GiB and reaches from_pretrained with no meta-tensor error. _build_meta_skeleton, its fallback warning, _resolve_init_config, the from_config resolution, and the accelerate init_empty_weights / infer_auto_device_map imports all went with it. Net 74 lines lighter.

Known approximation, worth flagging: total-bytes-vs-summed-GPU-budget ignores the per-device packing and no-split-module modelling infer_auto_device_map did. For a heuristic that only chooses whether to apply an 80% haircut, total size is the dominant term.

Tests cover the index path, the shard-sum fallback, a malformed index, capping only when the weights exceed the budget, and no cap when the size is unknown; the get_model tests also assert nothing is constructed during sizing.

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

Bot comment.

[P1] Preserve sizing for Hugging Face model IDs. This returns None whenever ckpt_path is not a local directory, but --pyt_ckpt_path explicitly accepts a Hugging Face model name and the README recommends passing the model card directly. Every org/model invocation therefore bypasses the headroom logic below; a large Hub model that device_map=auto spills to CPU can again fill GPU memory and OOM during calibration. Please resolve or download the small checkpoint index, or query Hub file metadata/cache, before returning unknown, and add a repository-ID test. Reserve the None fallback for genuinely unreadable layouts.

Correct, and it is one of the reasons the sizing logic is now gone entirely in 1b6a79d.

You were right about the gap: --pyt_ckpt_path accepts a model card (its help says so, and the README uses openai/gpt-oss-20b at line 245), hf_ptq.py:578 passes it straight into get_model unresolved, and _checkpoint_size_bytes bailed on anything that was not a local directory. Every hub-ID run silently lost the guard.

Rather than resolve hub metadata, the cap itself has been removed. It was only ever wanted in the offload case, where it is close to free — the run is already streaming weights, and the headroom is required because at 100% packed calibration OOMs immediately. Applied to a model that does fit it would convert a fully resident load into an offloaded one, which is a large certain slowdown introduced where there was none. That is why the original condition fired only on spill, and knowing which side you are on is exactly what needs a constructed model — the thing NVBug 6563509 is about.

So device_map="auto" now sizes itself; from_pretrained runs its own infer_auto_device_map on the real model regardless (transformers/modeling_utils.py:1362). --use_seq_device_map is the supported answer to GPU OOM, and --gpu_max_mem_percentage applies there and to --offload_folder — which is what its help text already said: "the percentage of available GPU memory to use for loading the model when device_map is set to sequential."

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

Bot comment.

[P1] Preserve per-device placement when deciding whether to reserve calibration headroom. Total checkpoint bytes fitting within the sum of GPU memory does not mean Accelerate can keep the model off CPU: parameters are indivisible and placement is per device. For example, two 9.5 GiB leaf parameters total 19 GiB across two 10 GiB GPUs, but infer_auto_device_map places one on GPU 1 and one on CPU, leaving the used GPU 95% full. This aggregate check reports that case as fitting and omits max_memory precisely when calibration may OOM. Please use a placement-aware or tensor-aware check, or a conservative condition that cannot produce this false fit, and add a regression test.

The principle is right and the code is gone as of 1b6a79d, but the suggested direction would have been wrong, and it is worth recording why.

On the principle: yes, an aggregate byte count cannot reproduce per-device placement, since parameters are indivisible. (The worked example does not hold — two 9.5 GiB parameters across two 10 GiB GPUs place one each and both fit. Three 6 GiB blocks on two 10 GiB GPUs is the real shape: 18 GiB aggregate against 20 GiB, yet the third spills.)

On the fix: a "conservative condition that cannot produce a false fit" would have been a regression. The cap is only wanted when the model is already offloading, where it is close to free and the headroom is required — at 100% packed, calibration OOMs immediately. Applied to a model that does fit, it converts a fully resident load into an offloaded one: a large, certain slowdown introduced where there was none. So a false "does not fit" is the more expensive error here, and biasing the check toward capping trades a rare loud OOM the user can fix with a flag for a guaranteed slowdown across a whole band of models.

Since the check must not be biased in either direction, and since answering it accurately requires constructing the model — which is the whole of NVBug 6563509 — the cap has been removed instead. device_map="auto" sizes itself; from_pretrained runs its own infer_auto_device_map on the real model anyway. --use_seq_device_map is the supported answer to GPU OOM.

No regression test, because there is no longer a condition to regress.

Superseded by 701180e — this code is back to its original form on main.

example_utils.py and its tests are restored to main except for the Phi-specific multimodal-detection heuristics. The meta-device skeleton, infer_auto_device_map, and the --gpu_max_mem_percentage cap all keep their original behavior, so there is nothing left in this thread to act on.

Reasoning: the cap is wanted exactly where it already fires — when the model is already offloading to CPU, where it costs little and the headroom is required. Every alternative explored on this branch (a more permissive skeleton, sizing from the checkpoint index, an unconditional budget) either changed behavior for models that fit today or moved the guard somewhere it does not belong, and all of it was for a crash that only ever affected the checkpoints this PR removes. NVBug 6563509 is resolved by dropping Phi-4-multimodal, not by changing the loader.

Can you please update this part to drop ph4mm?